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Abstract 

The  ARL  MSRC  has  conducted  a major  overhaul  of 
its  customer  service  process  in  order  to  better  support  our 
customer  community.  Users  were  asked  what  changes 
they  would  like  to  see  and  an  outside  consultant  was 
brought  in  to  take  a fresh  look  at  our  customer  service 
approach.  A new  methodology  for  providing  customer 
service  was  designed  to  overcome  known  deficiencies  in 
the  previous  system  as  well  as  incorporate  the  inputs  from 
our  users  and  the  consultant. 

We  purchased  new  hardware  (a  Sun  Fire  VI 00)  and 
software  (Remedy  5.1  Help  Desk)  to  implement  our  new 
customer  service  methodology.  Our  consultant  advised  us 
on  how  to  configure  and  use  Remedy  to  our  best 
advantage.  We  configured  Remedy  with  many  features  to 
allow  the  staff  to  take  direct  ownership  of  a user  problem 
report,  track  recurring  questions,  and  follow  the  tickets 
through  until  an  acceptable  solution  has  been  found.  One 
major  change  is  that  now  the  majority  of  the  staff  is  using 
Remedy  instead  of  just  the  front-line  Help  Desk.  The 
consultant  motivated  the  staff  on  the  importance  of 
providing  quality  customer  service. 

This  new  approach  to  customer  service  became 
operational  in  May  2003.  Since  then  we  have  collected 
various  statistics  on  the  types  of  problem  users  are 
encountering  so  we  can  improve  our  center.  We  use  this 
information,  along  with  customer  and  staff  feedback,  to 
continue  improving  the  system  to  meet  our  customer’s 
requirements. 

1.  Overview 

Over  the  past  two  years,  the  ARL  MSRC  has 
undergone  a major  transformation  in  the  way  we  perform 
customer  service.  We  recognized  that  the  old  process  was 
flawed,  and  needed  changes.  The  customers  are  the  most 
important  part  of  a major  High  Performance  Computing 
center  such  as  ARL.  Without  them,  the  systems  would  sit 
idle  and  waste  taxpayer  money.  With  them,  cutting-edge 
research  is  being  performed  daily  to  better  equip  the  US 


soldier.  The  ARL  MSRC  is  committed  to  helping 
researchers  and  scientists  compute  more  efficiently  and  to 
limit  the  amount  of  downtime  they  incur  as  a result  of 
system  or  application  failures.  We  realize  our  customers 
have  alternatives  for  performing  their  computing 
elsewhere  and  we  strive  to  make  every  customer’s 
interaction  with  us,  from  account  setup  to!  job  submission, 
as  easy  and  effective  as  possible. 

The  goals  of  this  transformation  in  customer  service 
are  to  provide  better  overall  service  to  the  customer  when 
they  contact  us,  either  through  email,  the  web,  or  a phone 
call.  We  want  to  make  sure  everyone  is  satisfied  with  the 
first  answer  they  receive,  and  that  they  receive  a correct 
solution  in  a timely  manner.  We  want  to  be  proactive  and 
resolve  as  many  issues  as  we  can  before  the  customer 
even  notices  that  something  may  have  gone  wrong.  The 
customer  should  notice  an  improvement  in  the  turn- 
around time  when  a help  request  is  submitted,  as  well  as  a 
reduction  in  the  number  of  times  they  haye  to  ask  for  help 
because  of  improved  processes  on  our  end. 

Customers  have  always  been  able  to  submit  problem 
reports  via  e-mail,  the  Web,  or  by  phone.  That  is  still 
true,  but  how  these  are  handled  now  is  very  different. 
Previously,  the  web  interface  was  just  a glorified 
formatting  engine  that  generated  an  e-mail  to  our  support 
staff.  When  an  e-mail  was  sent  to  the 
‘msrchelp@arl. army. mil’  e-mail  alias!  twenty-seven 
different  staff  people  received  it.  This  sometimes  resulted 
in  duplication  of  work  since  multiple  people  could  work 
on  the  same  ticket,  or  staleness  since  some  groups  would 
assume  other  groups  or  individuals  would  be  handling  a 
certain  ticket.  There  was  no  standard  process  for  keeping 
accurate  records  since  the  only  group  interacting  with 
Remedy  was  the  first  tier  Help  Desk.  Sometimes  no  one 
responded  right  away  and  other  times  multiple  people  did, 
showing  that  we  were  not  very  coordinated.  If  customers 
were  to  again  become  the  priority  of  the  ARL  MSRC,  this 
process  would  have  to  change  and  become  much  more 
organized. 
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The  ideas  first  presented  for  improvement  came  in  an 
internal  white  paper  written  by  Mr.  Steve  Thompson, 
Customer  Service  team  leader.  It  was  decided  that  while 
we  were  currently  using  the  Remedy  Helpdesk  system  to 
record  some  information  about  incoming  requests,  it  was 
not  being  used  anywhere  near  it’s  full  potential.  We  also 
wanted  to  limit  the  number  of  e-mails  that  our  staff  was 
receiving  so  there  would  not  be  duplicate  work  taking 
place,  and  they  would  not  have  to  read  about  issues 
unrelated  to  their  specialized  area.  We  wanted  to  make 
sure  our  staff  took  ownership  of  the  tickets  as  they  arrived 
and  worked  to  solve  them  in  a timely  manner.  We  also 
wanted  to  give  the  customer  the  opportunity  to  check  on 
the  status  of  his  or  her  ticket  through  the  web. 

2.  Developing  a Solution 

In  developing  a new  solution  to  improve  customer 
service  at  the  ARL  MSRC,  it  was  decided  that  a complete 
overhaul  of  the  hardware  and  software  was  needed.  The 
server  was  upgraded  from  an  old  Sun  Enterprise  3000  to  a 
new  Sun  Fire  V100.  Oracle  was  installed  as  the  backbone 
database  system  on  the  server  during  the  setup  process.  It 
was  decided  that  an  upgrade  from  an  earlier  version  of 
Remedy’s  Action  Request  System  was  needed  to  take 
advantage  of  the  latest  features.  Remedy’s  Action 
Request  System  5.1  was  installed  and  configured  over  a 
period  of  about  four  months  before  its  final  production 
release  was  made.  The  configuration  was  based  on 
Remedy’s  Help  Desk  System  component  with  major 
modifications  to  suit  the  needs  of  a large  HPC  center 
instead  of  a generic  customer  service  site. 

A major  part  of  the  configuration  of  the  new  Help 
Desk  system  was  the  development  of  a scheme  that 
classified  tickets  in  such  a way  that  made  the  routing  of  a 
ticket  automatic  and  the  general  information  about  a ticket 
available  at  a glance.  The  classification  scheme  consists 
of  a Category,  Type,  Item,  Service  and  Summary  (CTISS) 
that  allow  the  aforementioned  routing  and  quick 
assessment  of  a ticket  possible.  In  addition,  each  CTISS 
has  a set  priority,  assignment  group,  contact  time  and 
resolution  time  associated  with  it.  This  scheme  was 
developed  by  our  customer  service  team  in  conjunction 
with  an  outside  consultant,  Mr.  Joel  Ramseyer  of  The 
Diagonal  Group. 

Once  the  classification  scheme  was  decided  upon, 
configuration  and  development  of  the  Help  Desk  system 
in  Remedy  was  initiated.  The  development  and 
configuration  of  this  system  took  approximately  six 
weeks  of  dedicated  work.  Many  of  the  features  Remedy 
shipped  with  their  Help  Desk  configuration  were  not 
useful  and  were  removed.  The  remaining  features  were 
modified  to  work  with  the  process  flow  that  was  being 
created.  The  system  was  configured  in  such  a way  as  to 


allow  for  the  storage  of  ticket  resolutions  for  future  use 
and  the  ability  to  link  related  tickets  so  our  staff  only  has 
to  solve  the  problem  once  to  propagate  a valid  resolution 
to  all  interested  parties. 

Part  of  the  configuration  of  Remedy  was  the 
development  of  a web  interface  that  allows  customers  to 
not  only  submit  their  problem  via  the  web,  but  also  to 
classify  the  tickets.  By  classifying  the  problem  into  a 
CTISS,  Remedy  is  then  able  to  route  the  ticket  to  the 
correct  team  at  the  ARL  MSRC.  The  web  development  is 
based  on  Remedy’s  Action  Request  System  web  interface 
technology  and  it  allows  direct  submission  of  new  tickets 
into  Remedy.  In  addition  to  submitting  tickets  online, 
customers  were  also  given  the  capability  to  review  their 
submitted  tickets  online.  To  enforce  security  on 
information  about  their  submitted  tickets,  a wrapper  for 
the  page  was  integrated,  with  modifications,  to  meet  the 
ARL  MSRC  needs.  The  wrapper  to  authenticate 
customers  using  Kerberos  and  SecurlD  was  written  by 
Mr.  Vem  Staats  at  the  ASC  MSRC  and  has  proven  to 
work  extremely  well  in  this  environment. 

Once  the  development  and  configuration  of  the  client 
system  and  the  ticket  classification  scheme  were 
complete,  members  of  different  teams  at  the  ARL  MSRC 
tested  the  system  by  creating  sample  tickets  and  trying  to 
find  ways  to  disrupt  the  process  flow.  Once  the  bugs  that 
this  testing  uncovered  were  fixed,  the  system  was  ran  in 
parallel  with  the  existing  Help  Desk  system  to  see  if  any 
discrepancies  existed  that  were  not  found  during  the 
previous  test  runs.  Several  more  issues  were  found  and 
corrected  over  several  weeks.  Then  it  was  determined 
that  the  new  system  was  sufficiently  stable  to  support  a 
production  environment.  The  web  interface  took  longer 
to  develop  and  implement  because  of  security  issues.  The 
testing  of  this  tool  included  several  beta  testers  from  the 
customer  community  submitting  and  reviewing  tickets 
until  this  was  deemed  stable  as  well. 

3.  Implementation 

Once  all  the  development  work  was  completed,  the 
next  step  was  to  train  our  entire  local  staff  on  how  to  use 
the  new  Help  Desk  system  and  what  the  new  customer 
service  process  was  to  ensure  timely  and  accurate 
resolutions  to  problems.  A presentation  was  given  to  our 
staff  as  to  why  the  changes  were  made,  how  the  new 
process  works,  and  what  would  be  expected  from  them. 
This  included  a step-by-step  walkthrough  of  how  to 
interact  with  Remedy,  what  it  is  capable  of  doing,  and 
what  must  be  done  to  ensure  that  there  would  be 
accountability  for  each  ticket,  by  either  a team  or  an 
individual. 

As  mentioned  earlier,  the  old  Help  Desk  system  and 
the  new  Remedy  system  were  run  in  parallel  for  several 
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weeks.  This  was  done  to  ensure  the  new  system  was 
responding  properly  and  to  have  a backup  in  case  of  a 
catastrophic  failure,  which  fortunately  never  happened. 
The  old  system  was  then  phased  out  and  all  the  old 
records  archived  and  moved  into  the  new  Remedy  system 
in  case  any  historical  data  was  needed  in  the  future. 

As  the  old  system  was  phased  out,  and  the  new 
system  was  being  phased  in,  the  initial  web  interface  for 
submitting  tickets  was  rolled  out.  All  current  ARL 
MSRC  customers  were  informed  of  the  improvements  in 
the  customer  service  process.  Later,  as  the  interface  to 
review  tickets  was  completed  and  the  security  features 
were  thoroughly  tested,  the  new  interface  was  rolled  out 
on  the  site.  The  original  e-mail  had  made  note  that  this 
feature  was  coming.  It  was  linked  below  the  submission 
link  on  the  ARL  MSRC  Customer  Service  main  web 
page,  and  began  receiving  hits  that  day. 

4.  Noted  Improvements 

Our  complete  customer  service  overhaul  produced 
several  benefits.  All  web-based  ticket  submissions  are 
automatically  routed  to  the  team  that  is  best  suited  for  the 
problem  based  on  the  CTISS  entered.  If  no  CTISS  is 
entered,  or  the  ticket  is  submitted  using  e-mail  or  a phone 
call,  our  first-level  Help  Desk  will  categorize  the  ticket  as 
it  is  entered  into  the  system,  and  the  ticket  will  be  routed 
from  there.  This  way,  only  the  right  staff  people  are 
notified,  and  others  do  not  waste  time  reading  about 
problems  unrelated  to  their  work. 

Tickets  migrate  through  various  stages  as  they  are 
worked.  A ticket  is  considered  ‘Assigned’  once  it  is  in 
the  Remedy  system  and  assigned  to  a team.  Once  a team 
acknowledges  that  they  are  the  proper  team  to  work  on  a 
ticket,  it  is  placed  in  an  ‘Accepted’  status.  When  a ticket 
is  being  actively  worked,  there  is  a ‘Work  In  Progress’ 
status  that  lets  other  members  of  the  team  know  who  is 
actively  working  the  issue.  A ‘Research’  status  indicated 
that  there  is  some  outside  factor  preventing  this  ticket 
from  being  actively  worked,  such  as  waiting  for  a vendor 
or  customer  response.  When  the  individual  working  the 
ticket  believes  it  is  fixed,  he  places  it  in  a ‘Resolved’ 
status  and  Remedy  provides  a convenient  mechanism  for 
him  to  compose  a resolution  e-mail  to  the  customer. 
When  a customer  is  satisfied  that  the  resolution  is 
accurate  and  complete,  he  or  she  can  ‘Close’  the  ticket.  If 
not,  the  customer  can  re-open  the  ticket.  If  no  response  is 
received  with  one  week,  Remedy  automatically  closes 
tickets  in  the  ‘Resolved’  state. 

To  ensure  that  these  rules  are  followed,  there  is  a 
limit  as  to  how  long  a ticket  can  be  placed  into  a 
‘Research’  status,  and  reminders  are  e-mailed  to  team 
members  if  a ticket  goes  unaccepted  for  a specific  period 
of  time.  If  a ticket  is  not  resolved  in  a timely  manner,  it  is 


escalated  to  team  leaders  and  management  to  let  them 
know  that  there  has  been  a delay  in  the  process 
somewhere  along  the  line.  The  front-line  Help  Desk  has 
responsibility  to  check  periodically  on  stale  and 
‘Research’  tickets  to  ensure  that  they  have  not  been 
forgotten  and  that  a resolution  is  being  worked  on. 

The  new  Help  Desk  system  and  processes  have  been 
in  place  for  over  a year  now.  In  the  year  of  data  available 
at  this  writing  (May  2003  through  April  2004),  there  have 
been  periods  of  improvement  and  periods  that  did  not 
meet  our  expectations.  From  it’s  inception  last  May 
through  the  fall,  our  overall  success  rate  in  resolving 
tickets  on  time  met  or  exceeded  our  goal  of  95%.  As  this 
period  progressed,  tickets  were  broken  down  by  priority 
and  by  team  to  see  where  improvements  were  still  needed. 
Over  time  we  also  tweaked  the  system  for  more  realistic 
expectations  and  to  correct  routing  flaws.  By  not 
understanding  explicitly  how  certain  processes  worked, 
the  resolution  time  for  some  types  of  tickets  were  not 
achievable  and  needed  to  be  modified.  Oyer  the  winter, 
we  failed  to  meet  our  goals  because  of  some  staff 
shortages  which  have  since  been  addressed.  Since  new 
members  have  joined  our  staff  and  come  up  to  speed  on 
our  procedures,  our  ticket  resolution  rates  are  improving. 
For  April  2004,  only  one  request  for  help  from  a customer 
was  not  responded  to  in  a timely  manner. 
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Figure  1.  On-Time  Ticket  Resolutions 

Currently  we  feel  the  new  process  that  has  been 
developed  is  working,  and  customer ! service  has 
improved.  During  April,  we  reached  an  on-time  ticket 
resolution  rate  of  99.6%,  our  best  thus  far.  While  still 
searching  for  a perfect  month  resolving  customer 
problems,  steps  have  been  taken  to  keep  the  number  of 
tickets  not  resolved  on-time  as  low  as  possible.  New 
members  have  been  added  to  the  first-level  Help  Desk 
team,  and  the  Application  Support  team.  Also,  since  new 
software  installations  and  upgrades  have  been  a persistent 
problem,  we  hired  a dedicated  software  install  engineer  to 
coordinate  and  perform  the  installations.  1 This  should 


reduce  the  number  of  low  priority  tickets  that  are  not 
resolved  in  a timely  manner. 

5.  Future  Plans 

For  the  future,  the  ARL  MSRC  is  researching  the 
latest  incarnation  of  the  Remedy  Action  Request  system, 
version  6.0.  It  is  anticipated  that  there  will  be  an  upgrade 
this  year  to  that  version  once  some  of  the  initial  bugs  are 
worked  out  and  it  appears  to  be  stable  enough  for  our 
production  environment.  An  upgrade  of  the  Remedy 
system  would  not  require  a complete  re-implementation  at 
this  point,  but  it  would  require  another  phase  of  testing 
and  possibly  some  new  development  to  take  advantage  of 
the  latest  features. 

In  addition  to  the  upgrade  of  the  Remedy 
environment,  there  are  new  features  which  are  currently 
being  implemented,  tested  and  considered.  Development 
is  continuing  on  a set  of  tools  to  gather  the  metrics  needed 
to  make  accurate  judgments  about  how  our  customer 
service  team  is  doing  under  this  new  process.  Also  under 
consideration  is  a new  feature  for  the  web-based  system 


which  will  pop-up  a ‘canned’  solution  if  certain 
conditions  are  met.  The  new  Remedy  6.0  environment 
will  offer  more  new  features  which  may  make  more 
enhancements  possible. 

Outside  of  the  Help  Desk  environment,  our  Customer 
Service  team  is  busy  spending  their  available  time 
working  on  solutions  that  will  help  to  prevent  problems 
from  occurring,  or  let  us  know  when  problems  occur  so 
we  can  fix  them  before  they  are  noticeable  to  the 
customers.  One  of  these  projects  is  to  improve  our  script 
generator  tool  GEST  to  support  more  commercial-off-the- 
shelf  (COTS)  packages  and  the  LSF  queuing  system.  We 
are  also  working  on  having  modules  available  on  all 
systems  for  all  COTS  packages  and  system  tools.  These 
tools  will  allow  for  easier  interaction  with  the  system  so 
that  the  influx  of  new  architectures  and  a new  queuing 
system  are  not  as  troublesome  as  they  otherwise  would 
be.  In  addition,  we  are  revamping  our  website  to  give  it 
an  updated  look  and  to  better  organize  the  information  so 
that  user  guides,  documentation  and  other  frequently 
accessed  data  is  easier  to  find. 
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